Reports from Other Evaluation Campaigns and Panel Discussion: TREC is 20 Years Old, Where Now for Evaluation Campaigns?

Although shared test collections have been around for since the 1960s, the arrival of TREC in 1992 was a major boost to information retrieval research. It spawned a wide range of evaluation campaigns around the world including NTCIR, CLEF, FIRE, INEX, etc. However, in recent years, the number of papers at major conferences, such as SIGIR or CIKM, that report research using the collections produced by these campaigns appear to be reducing. More papers are describing research using so-called private data sets. So, what will the evaluation campaigns look like in 20 years from now? Is the rise of private data a threat to the scientific integrity of IR research, or a valuable diversification of research focus? Speakers’ Bios Ian Soboroff is Acting Group Leader at NIST in Washington DC, USA. Gareth Jones is a senior lecturer at Dublin City University, Ireland. Andrew Trotman is a Senior Lecturer at the University of Otago, New Zealand. Shlomo Geva is an Associate Professor at Queensland University of Technology, Australia. Nicola Ferro is an Assistant professor in Computer Science at the Department of Information Engineering and at the Faculty of Statistical Sciences of the University of Padua, Italy. Hideo Joho is an Assistant Professor in the Faculty of Library, Information and Media Science at the University of Tsukuba. Mark Sanderson is a Professor in Information Retrieval at RMIT University, Melbourne, Australia. William Webber is a researcher at University of Maryland, USA. ― 24 ― The Fourth International Workshop on Evaluating Information Access (EVIA), December 6, 2011, Tokyo, Japan